---
id: datadog
title: Datadog metrics setup - Temporal Cloud feature guide
sidebar_label: Datadog
description: Learn to export Cloud metrics from Temporal Cloud to Datadog, enhancing observability to monitor, alert, and visualize your applications and infrastructure seamlessly.
slug: /cloud/metrics/datadog
toc_max_heading_level: 4
keywords:
  - temporal cloud datadog integration
  - exporting temporal cloud metrics
  - datadog observability integration
  - helm minikube setup
  - prometheus api temporal cloud
  - metrics monitoring datadog
  - datadog kubernetes integration
  - helm kubernetes deployment
  - kubernetes observability
  - datadog metrics dashboard
  - prometheus querying datadog
  - temporal cloud observability
  - helm setup minikube
  - monitoring temporal metrics
  - temporal cloud dashboards
tags:
  - Metrics
  - Observability
  - Temporal Cloud
---

Exporting cloud metrics to Datadog provides enhanced observability, allowing you to monitor, alert, and visualize key performance indicators of your applications and infrastructure.
Temporal's integration with Datadog extends the monitoring capabilities of your Temporal Cloud deployment.

:::note

This tutorial provides an example application to help users capture metrics from Temporal's PromQL endpoint and export to DataDog.

:::

### What will you learn?

You will set up your environment to export metrics from Temporal Cloud to Datadog, including:

- Preparing your environment with the necessary prerequisites
- Configuring certificates for secure communication
- Deploying the integration using Helm and Minikube
- Verifying the setup to ensure metrics are being exported correctly

## Prerequisites

Before you begin, ensure you have the following:

- A [Datadog account](https://www.Datadoghq.com/) with an [API Key](https://app.Datadoghq.com/account/settings#api).
- A [Temporal Cloud account](https://cloud.temporal.io/).
- Helm installed [for managing Kubernetes applications](https://helm.sh/docs/intro/install/).
- K9s for [Kubernetes CLI UI management](https://github.com/derailed/k9s).
- Minikube for running a [local Kubernetes cluster](https://minikube.sigs.k8s.io/docs/start/).
- A GitHub account [for accessing repositories](https://github.com/).

## Step 1. Set up the GitHub project

Temporal provides a script demonstrating the minimum work necessary to read recently generated metrics from a Temporal Cloud account using the Prometheus API and import them into Datadog while handling some common edge and error cases.

The following tutorial uses the [Prometheus Querying Language to Datadog in Go](https://github.com/temporalio/samples-server/tree/main/cloud/observability/promql-to-dd-go) (PromQL); however, you can use and customize this code and others to suit your needs.

Temporal also provides a [scrape version](https://github.com/temporalio/samples-server/tree/main/cloud/observability/promql-to-scrape) which can be used by the Otel or Datadog agents.

Temporal also provides examples in other languages:

- [PromQL to Datadog in Python](https://github.com/temporalio/samples-server/blob/main/cloud/observability/README.md)
- [PromQL to Datadog in Typescript](https://github.com/temporalio/samples-server/tree/main/cloud/observability/promql-to-dd-ts)

Clone the `promql-to-dd-go` directory from the Temporal samples server and navigate into it:

```bash
gh repo clone temporalio/samples-server
cd samples-server/cloud/observability/promql-to-dd-go
```

:::note

These examples are provided as-is, without support.
They are intended as reference material only.

:::

Metrics are exported on a per-account basis.

## Step 2. Create your certificates

Choose a method for generating certificates.
You can use one of the methods provided, or if your organization has its own certification manager, use that.

These certificates are used to communicate with your Temporal Cloud account.
For more information on certifications and Temporal Cloud, see [Certificate management](/cloud/certificates).

### Using tcld

Use the tcld to authenticate, generate, and add certificates to your Cloud account.

1. **Login**: Authenticate with your Temporal Cloud account.
   ```bash
   tcld login
   ```
2. **Create certs**: Generate the necessary certificates for secure communication.
   ```bash
   tcld generate-certificates certificate-authority-certificate --org ${ACCOUNT_ID} -d 1y --ca-cert ca.pem --ca-key ca.key
   ```
3. **Add certificates to Cloud account**: Ensure your Temporal Cloud Namespace is configured with the generated certificates.
   ```bash
   tcld namespace accepted-client-ca add --ca-certificate-file ca.pem
   ```

Replace `${ACCOUNT_ID}` with your Temporal Cloud account ID.
This is the assigned account identifier.

### Using certstrap

You can also use certification managers like certstrap.

1. Follow the [certstrap README](https://github.com/square/certstrap) to download and install certstrap.
2. Create a new certificate authority with `certstrap init --common-name CertAuth`.
3. Request a certificate key pair with `certstrap request-cert --common-name metrics-cert`.
4. Sign the certificate request to generate the end-entity certificate with `certstrap sign metrics-cert --CA CertAuth`.
5. Locate your newly created certificates in the `out` folder within the certstrap directory.
6. Add the certificates to your Cloud Account, for more information see [How to add, update, and remove certificates in a Temporal Cloud Namespace](/cloud/certificates#update-certificates-using-temporal-cloud-ui).

:::note

Certificate names must be unique per account.

:::

Next, verify that your certificates are set up correctly.

### Verifying certificates setup

Now that you've created and set your certifications, test them to ensure they are working correctly.

<Tabs>
  <TabItem value="input" label="Input" default>
Test your setup with the following command:
```bash
curl --cert ca.pem --key ca.key "https://<Organization_ID>.tmprl.cloud/prometheus/api/v1/query?query=temporal_cloud_v0_state_transition_count" | jq .
```

Provide the Organization ID of your Temporal Cloud account.

</TabItem>
<TabItem value="ouput" label="Output">
Expect a successful response indicating your setup is correct.

```json
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "temporal_cloud_v0_state_transition_count",
          "__rollup__": "true",
          "operation": "WorkflowContext",
          "temporal_namespace": "namespace.id",
          "temporal_service_type": "history"
        },
        "value": [123456.789, "12"]
      }
    ]
  }
}
```

</TabItem>
</Tabs>

## Step 3. Run Helm and Minikube

Use the following commands to run **Minikube** and **Helm** in your environment.
While this is using Minikube, you can deploy to any Kubernetes cluster, with your environment specific configuration.

1. **Start Minikube**:

In one terminal, start your **Minikube** instance:

```bash
minikube start
```

2. **Deploy with Helm**:

In a second terminal, start your **Helm** instance.

<Tabs>
  <TabItem value="input" label="Input" default>

Set your environment variables for the Datadog API key and [Temporal Account Id](/cloud/namespaces#temporal-cloud-account-id), then deploy using Helm:

```bash
helm install promqltodd . \
  --set prom_endpoint=https://${ACCOUNT_ID}.tmprl.cloud/prometheus \
  --set dd_api_key=${DD_API_KEY} \
  --set query_interval_seconds=15 \
  --set-file 'ca_cert=../out/metrics-cert.crt' \
  --set-file 'ca_key=../out/metrics-cert.key'
```

Update the path and the name of your certificate and key.

</TabItem>
<TabItem value="ouput" label="Output">

You'll see an output indicating successful deployment:

```bash
NAME: promqltodd
LAST DEPLOYED: [Deployment Time]
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing promql-to-dd-go.
```

</TabItem>
</Tabs>

Use the `helm status` command to confirm the deployment or `helm get all` to view the deployment.

### Deploy K9s

Start your **K9s** instance to monitor the deployment:

```bash
k9s
```

Review the logs to confirm the integration is functioning as expected.

Use a [pre-built dashboard](https://github.com/temporalio/samples-server/blob/main/cloud/observability/) to visualize your metrics in Datadog.

You've successfully set up the export of metrics from Temporal Cloud to Datadog.

### Next steps

This is just the start.

Customize and extend this code, dashboards, and deployment processes to meet your needs.

- Explore [Datadog dashboards](https://github.com/temporalio/samples-server/blob/main/cloud/observability/) to visualize your Temporal metrics.
- Set up alerts in Datadog based on the metrics received from Temporal Cloud.
- Consider integrating additional observability tools or exporting metrics to other platforms as needed.
